Method for controlling triggering of human-computer interaction operation and apparatus thereof

ABSTRACT

A method for controlling triggering of a human-computer interaction operation and an apparatus are provided according to the embodiments of the disclosure. The method includes: acquiring images shot by a shooting device, and displaying the acquired images on a display screen in a blurring manner; detecting a difference between acquired images in each frame, recognizing a designated outline on the image according to the difference between acquired images in each frame, and calculating a location of the recognized designated outline on the display screen; and determining, in real time, whether the designated outline at the location on the display screen intersects with a designated area displayed on the display screen, and triggering an operation corresponding to the designated area if the designated outline at the location on the display screen intersects with the designated area displayed on the display screen.

CROSS REFERENCE TO RELATED APPLICATIONS

The application is a continuation application of InternationalApplication PCT/CN2013/087811 filed on Nov. 26, 2013, titled “TRIGGERAND CONTROL METHOD AND SYSTEM FOR HUMAN-COMPUTER INTERACTION OPERATION”,which claims priority to Chinese Patent Application No. 201210583819.6,filed on Dec. 28, 2012 with the Chinese Patent Office, titled “METHODFOR CONTROLLING TRIGGERING OF HUMAN-COMPUTER INTERACTION OPERATION ANDAPPARATUS THEREOF”, both of which are incorporated herein by referencein their entireties.

TECHNICAL FIELD

The disclosure relates to the field of human-computer interactiontechniques of a computer, and in particular, to a method for controllingtriggering of a human-computer interaction operation and an apparatusthereof.

BACKGROUND

Human-computer interaction techniques refer to technologies foreffectively achieving human-computer dialogues by means of input devicesand output devices of a computer. These technologies include: providing,by a computer, a large amount of relevant information, prompts,requests, and the like for human by means of output devices or displaydevices, and inputting, by human, relevant information, operationinstructions, and answers for questions, or the like for the computer bymeans of input devices. The human-computer interaction techniques areplaying an important role in designing user interface of a computer.

In the conventional human-computer interaction techniques, in case thatrelevant information is input by human into a computer by means of inputdevices, operations are generally performed by hands. For example, theinput device may be a keyboard, a mouse or a touch screen, etc. Relevantinstruction information is input by human by using a keyboard, and thecomputer may respond to the instruction information and perform acorresponding operation; alternatively, a relevant button on a computerinterface may be clicked by human by using a mouse to input theinstruction, and the computer may respond to the instruction and performa corresponding operation. For example, a “close” button is clicked byhuman by using a mouse, and the computer closes a window correspondingto the “close” button.

SUMMARY

A method for controlling triggering of a human-computer interactionoperation and an apparatus thereof are provided according to embodimentsof the disclosure, to facilitate the disabled to trigger a computeroperation in a contactless manner.

A method for controlling triggering of a human-computer interactionoperation is provided, which includes:

acquiring images shot by a shooting device, and displaying the acquiredimages on a display screen in a blurring manner;

detecting a difference between the acquired images in each frame,recognizing a designated outline on the acquired image according to thedifference between the acquired images in each frame, and calculating alocation of the recognized designated outline on the display screen; and

determining, in real time, whether the designated outline at thelocation on the display screen intersects with a designated areadisplayed on the display screen, and triggering an operationcorresponding to the designated area if the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen.

An apparatus for controlling triggering of a human-computer interactionoperation is provided, which includes at least one processor and amemory having processor-executable instructions stored therein, and theinstructions when executed by the at least one processor, configure theapparatus to:

acquire images shot by a shooting device, and display the acquiredimages on a display screen in a blurring manner;

detect a difference between the acquired images in each frame, recognizea designated outline on the image according to the difference betweenthe acquired images in each frame, and calculate a location of therecognized designated outline on the display screen; and

determine, in real time, whether the designated outline at the locationon the display screen intersects with a designated area displayed on thedisplay screen, and trigger an operation corresponding to the designatedarea if the designated outline at the location on the display screenintersects with the designated area displayed on the display screen.

It can be seen that in the embodiments of the disclosure, the imagesshot by the shooting device are acquired; the images are displayed onthe display screen in the blurring manner such as in a translucentmanner, so that the images may superimpose other interface displayed onthe display screen, and the designated outline (for example, an outlineof an organ such as an eye of human or a mouth of human) on the imagecan be recognized; and a user may control movement of the designatedoutline on the image by moving his body, and in case that the designatedoutline intersects with the designated area (which, for example, may bea display area for media information, or a designated instruction areasuch as a button or a link, etc) displayed on the display screen, theoperation corresponding to the designated area is triggered. Therefore,the human-computer interaction operation may be triggered without usinghands, which facilitates those with disabled hands to trigger a computeroperation in a contactless manner.

BRIEF DESCRIPTION OF THE DRAWINGS

The following drawings are some embodiments of the technical solutionsof the present disclosure, and embodiments of the present disclosure arenot limited to the features shown in the drawings. In the followingdrawings, similar reference signs represent similar elements.

FIG. 1 is a schematic flowchart of a method for controlling triggeringof a human-computer interaction operation according to an embodiment ofthe disclosure;

FIG. 2 a is a schematic diagram of a first computer interface on which adesignated area displayed on a display screen is a display area ofdesignated media information according to an embodiment of thedisclosure;

FIG. 2 b is a schematic diagram of a second computer interface on whicha designated area displayed on a display screen is a display area ofdesignated media information according to an embodiment of thedisclosure;

FIG. 3 a is a schematic diagram of a first computer interface on which adesignated area displayed on a display screen is a designatedinstruction area according to an embodiment of the disclosure;

FIG. 3 b is a schematic diagram of a second computer interface on whicha designated area displayed on a display screen is a designatedinstruction area according to an embodiment of the disclosure;

FIG. 4 is a schematic composition diagram of an apparatus forcontrolling triggering of a human-computer interaction operationaccording to an embodiment of the disclosure; and

FIG. 5 is a schematic hardware diagram of an apparatus for controllingtriggering of a human-computer interaction operation according to anembodiment of the disclosure.

DETAILED DESCRIPTION OF EMBODIMENTS

The disclosure is further described below in detail in combination withthe drawings and specific embodiments.

Throughout the descriptions of the embodiments, the term “shootingdevice” may alternately be described as “an image acquisition device”(such as a fixed image camera or a video camera).

For simple and intuitive description, the solutions of the disclosureare illustrated below by describing some representative embodiments. Alarge number of details in the embodiments are merely used to helpunderstanding the solutions of the disclosure. Obviously, implementationof the technical solutions of the disclosure may not be limited to thesedetails. In order to avoid unnecessarily obscuring the solutions of thedisclosure, some embodiments are not described in detail, and onlyframeworks are given in these embodiments. In the following, “include”refers to “include, but is not limited to”, and “according to . . . ”refers to “at least according to . . . ”, but is not limited to “onlyaccording to . . . ” Due to Chinese customary language usage,hereinafter in case if a certain type of component is not defined, itmay be understood that there may be at least one or more components ofthis type.

Currently, it is widely accepted that a human-computer interactionoperation is performed by hands, however, for those with disabled hands,the technology, in which information and instruction are input into acomputer by operations of hands and thus the human-computer interactionoperation can not be achieved in a contactless manner, is inappropriate.Although there is a technical solution, in which contactless input inhuman-computer interaction is performed by hand gestures, butcorresponding actions need to be performed by hands in this technicalsolution, which is still inconvenient for those with disabled hands.

A method for controlling triggering of a human-computer interactionoperation and an apparatus thereof are provided according to embodimentsof the disclosure. In the embodiments of the disclosure, images shot bya shooting device are acquired, and the images are displayed on adisplay screen in a blurring manner; difference between the images ineach frame is detected, a designated outline on the shot image isrecognized according to the difference between the images in each frame,and a location of the recognized designated outline on the displayscreen is calculated; and whether the designated outline at the locationon the display screen intersects with a designated area displayed on thedisplay screen is determined in real time, and an operationcorresponding to the designated area is triggered if the designatedoutline at the location on the display screen intersects with thedesignated area displayed on the display screen. In the embodiments ofthe disclosure, the human-computer interaction operation may betriggered without using hands, which facilitates those with disabledhands to trigger a computer operation in the contactless manner.

FIG. 1 is a schematic flowchart of a method for controlling triggeringof a human-computer interaction operation according to an embodiment ofthe disclosure. As shown in FIG. 1, the method mainly includes step 101to step 104.

Step 101 includes: acquiring images shot by a shooting device, anddisplaying the images on a display screen in a blurring manner.

The blurring manner may be a designated display manner, such as atranslucent display manner. For example, the images are displayed on thedisplay screen in the translucent manner; or the images may be convertedinto animated outline images (for example, animated images with simpleoutlines), and the animated outline images may be superimposed to anoriginal interface of the display screen, so that a user may see theoriginal interface of the display screen, and the animated outlineimages, which facilitates the user to perform subsequent operations bymoving the images. In the following embodiments, the case that theimages are displayed on the display screen in the translucent manner istaken as an example for illustration.

Step 102 includes: detecting difference between images in each frame,recognizing a designated outline on the shot image according to thedifference between images in each frame, and calculating a location ofthe recognized designated outline on the display screen.

Step 103 includes: determining, in real time, whether the designatedoutline at the location on the display screen intersects with adesignated area displayed on the display screen, and proceeding to step104 if the designated outline at the location on the display screenintersects with the designated area displayed on the display screen, andreturning to step 102 if the designated outline at the location on thedisplay screen does not intersect with the designated area displayed onthe display screen.

Step 104 includes: triggering an operation corresponding to thedesignated area.

In this embodiment of the disclosure, the recognizing a designatedoutline on the image may be recognizing an outline of a human organ, ormay alternatively be recognizing an outline of other pattern. Generally,a shooting device (such as a camera) installed in an apparatus such as acomputer shoots a head of a user. For example, in case that the userperforms a video chat by using a video chatting tool, the shootingdevice generally shoots images of the head of the user, especially,facial images. Therefore, in order to facilitate operations of users,especially, those with disabled hands, in an embodiment of thedisclosure, the recognizing a designated outline on the image may berecognizing an outline of a human eye, since the human eye has astandard outline, and further operation instructions may be sent to acomputer by further detecting a motion form such as a blink.

The recognized designated outline on the image may alternatively be anoutline of a human organ such as a mouth, and even may alternatively bea designated standard pattern. For example, a white board painted withthe designated pattern may be provided for the user in advance, the usermay lift the white board in front of the camera so that the camerashoots the designated pattern (such as an ellipse with a clear outline)on the white board, and the designated pattern is the designated outlineto be detected in this embodiment of the disclosure. In case that theuser moves the white board so that the designated pattern displayed onthe display screen intersects with the designated area (which, forexample, may be a display area of media information, or a designatedinstruction area such as a button or a link, etc), the operationcorresponding to the designated area is triggered. In this way, thehuman-computer interaction operation may be achieved without usinghands, thereby facilitating those with disabled hands to trigger acomputer operation in a contactless manner.

The case that the designated outline is an outline of an eye is taken asan example for illustration in the following embodiments of thedisclosure.

In step 102, the process of detecting difference between images in eachframe, recognizing a designated outline on the shot image according tothe difference between images in each frame, and calculating a locationof the recognized designated outline on the display screen may beachieved by using a programming tool, for example, by using a targetedinterface function in an open source computer vision library (openCV).

The OpenCV is a cross-platform computer vision library distributed basedon an open source, and may run on a computer operating system such asLinux, Windows, or Mac OS. The OpenCV is lightweight and efficient, isformed by a series of C functions and a small number of C++ classes,provides invocation interfaces of languages such as Python, Ruby, andMATLAB, and implements many universal calculation methods in imageprocessing and computer vision.

For example, in an embodiment, the difference between the images in eachframe may be detected by using a cvSub interface function and acyThreshold interface function in the OpenCV. For example, instructionsof specific implementation code are as follows:

-   -   cvSub(gray, prev, diff, NULL);    -   cvThreshold(diff, diff, 5, 255, CV_THRESH_BINARY).

The gray is a current frame of the image, the prev is a previous frameof the current frame, and the diff is the difference between images ineach frame.

For example, in an embodiment, the designated outline may be recognizedaccording to the difference between images in each frame by using acvFindContour interface function in the OpenCV, for example, recognizingthe eye outline. For example, specific instructions of implementationcode are as follows:

int nc = cvFindContours( diff, /* the difference image */ storage, /*created with cvCreateMemStorage( ) */ &comp, /* output: connectedcomponents */ sizeof(CvContour), CV_RETR_CCOMP, CV_CHAIN_APPROX_SIMPLE,cvPoint(0,0) ).

The diff is the calculated difference between images in each frame; thecomp is the recognized eye outline, and the eye outline is output by thecvFindContour interface function.

For example, in an embodiment, the location of the recognized designatedoutline on the display screen may be calculated by using a cvSetImageROIinterface function in the OpenCV, a specific implementation codeinstruction of which is as follows:

-   -   cvSetImageROI(gray, rect_eye).

The rect_eye is a location of the eye outline output by the interfacefunction cvSetImageROI on the image in the current frame gray, and acurrent location of the eye outline on the display screen may becalculated according to a location occupied by the image in the currentframe gray on the display screen.

In this embodiment of the disclosure, the designated area displayed onthe display screen may have multiple forms. For example, the designatedarea may be a display area of electronic media information (referred asmedia information in the embodiments of the disclosure), and mayalternatively be the designated instruction area such as a designatedbutton, a designated word link, a designated picture area, etc.

FIG. 2 a is a schematic diagram of a first computer interface on which adesignated area displayed on a display screen is a display area ofdesignated media information. As shown in FIG. 2 a, media information201 and media information 202 are displayed on the computer interface200. In this embodiment of the disclosure, a head of a user shot by ashooting device is displayed on the computer interface 200 in atranslucent manner. In this way, the head of the user may superimposeinformation on the computer interface 200, so that the user may not onlysee a variety of information on the interface 200 clearly, but also seehis head. Thus the user may observe movement of the eye outline whilemoving his head, so that the eye outline can move to the designatedarea, i.e., a display area of the media information 201 or a displayarea of the media information 202. FIG. 2 b is a schematic diagram of asecond computer interface on which a designated area displayed on adisplay screen is a display area of designated media information. Asshown in FIG. 2 b, in case that an eye outline 203 of a user moves tothe display area of the designated media information 201, that is, theeye outline 203 on the display screen intersects with the display areaof the media information 201, an operation corresponding to the displayarea of the designated media information 201 is triggered.

In an embodiment, in case that the designated outline such as the eyeoutline 203 on the display screen intersects with the designated areadisplayed on the display screen, such as the display area of the mediainformation 201, the triggered operation corresponding to the displayarea of the designated media information 201 includes: recording aperiod of time in which the eye outline 203 intersects with the displayarea of the designated media information 201, determining whether theeye outline 203 on the display screen moves out of the display area ofthe designated media information 201, stopping recording the period oftime if the eye outline 203 on the display screen moves out of thedisplay area of the designated media information 201, and continuingrecording the period of time if the eye outline 203 on the displayscreen does not move out of the display area of the designated mediainformation 201. In this way, degree of concern of the user on the mediainformation 201 may be calculated based on the period of time, and otherrelevant operation such as a billing operation may be further performedaccording to the degree of concern. That is, billing informationcorresponding to the designated media information 201 is calculatedaccording to the recorded period of time in which the designated outline(such as the eye outline 203) intersects with the display area of thedesignated media information 201.

Generally, for media information (for example, web advertisements are akind of media information) presented on networks, billing is performedbased on clicks of users on the media information and the number oftimes of exposures of the media information, rather than based on lengthof time that visitors view the media information. According to aboveprocessing steps in the embodiment of the disclosure, the period of timethat the eye outline of the user intersects with the designated mediainformation, equivalent to degree of concern of the user on the mediainformation, may be calculated, on the basis of which a new way forbilling the media information may be achieved. For example, as shown inFIG. 2 b, timing is started once the visitor moves his head to make aneye outline 203 of the visitor displayed on the display screen intersectwith the display area of the designated media information 201, timing isstopped once the eye outline 203 on the display screen moves out of thedisplay area of the designated media information 201, and billing on themedia information 201 is started if the period of time exceedspredetermined number of seconds, so that the media information may bebilled based on the degree of concern of the user on the mediainformation, and thus the billing may be more specific and moreaccurate.

In another embodiment, in case that the designated outline on thedisplay screen intersects with the designated area displayed on thedisplay screen, the triggered operation corresponding to the displayarea of the designated media information includes: detecting whether adesignated motion form of the designated outline occurs, and triggeringan instruction operation bound with the designated area if thedesignated motion form of the designated outline occurs. For example,whether the eye outline 203 blinks is detected, and the instructionoperation bound with the designated area is triggered if the eye outline203 blinks. For example, the instruction operation bound with thedesignated media information 201 is a click. The click on the mediainformation 201 may be triggered after the user blinks, thereby openinga web page to which the media information 201 is directed.

FIG. 3 a is a schematic diagram of a first computer interface on which adesignated area displayed on a display screen is a designatedinstruction area. As shown in FIG. 3 a, media information 201 and mediainformation 202 are displayed on the computer interface 200. There isalso a designated instruction area on the media information 201. Forexample, a “next” button 301 and a “close” button 302 are bothdesignated instruction areas. The instruction operation bound with the“next” button 301 is to switch the current media information 201 to nextmedia information, and an instruction operation bound with the “close”button 302 is to close the current media information 201. In thisembodiment of the disclosure, a head of a user shot by a shooting deviceis displayed on the computer interface 200 in a translucent manner. Inthis way, the head of the user may superimpose information on thecomputer interface 200, so that the user may not only see a variety ofinformation clearly, but also see his head, on the interface 200. Thusthe user can observe movement of the eye outline while moving his head,so that the eye outline can move to the designated instruction area.

FIG. 3 b is a schematic diagram of a second computer interface on whicha designated area displayed on a display screen is a designatedinstruction area. In case that an eye outline 203 of a user moves to the“next” button 301, that is, the eye outline 203 intersects with the“next” button 301, whether a designated motion form (such as blinking)of the eye outline 203 occurs is detected. If the designated motion formof the eye outline 203 occurs, an instruction operation bound with the“next” button 301 is triggered, that is, the current media information201 displayed on the display area of the media information is switchedto next media information. In case that the eye outline 203 of the usermoves to the “close” button 302, that is, the eye outline 203 intersectswith the “close” button 302, whether a designated motion form (such asblinking) of the eye outline 203 occurs may be detected. If thedesignated motion form of the eye outline 203 occurs, an instructionoperation bound with the “close” button 302 is triggered, that is, thecurrent media information 201 is closed.

In other embodiment, in case that the designated outline is an outlineof other pattern, the designated motion form may be an actioncorresponding to the outline of other pattern. For example, in case thatthe designated outline is an outline of a human mouth, the designatedmotion form may be opening and closing actions of the mouth.

In an embodiment, the process of detecting whether a designated motionform of the designated outline occurs includes:

creating a template of the designated outline, for example, in anembodiment, creating an eye template by using a cvResetImageROI(gray)interface function in an OpenCV;

detecting images in each frame in the template (such as the eyetemplate) of the designated outline, determining whether change betweenthe images in each frame satisfies the designated motion form andtriggering the instruction operation bound with the designated area ifthe change between the images in each frame satisfies the designatedmotion form.

For example, in case that the designated outline is the eye outline, theprocess of detecting whether a designated motion form of the designatedoutline occurs may include: detecting whether the eye outline blinks.

A specific method of detecting whether the outline of an eye blinks mayinclude: detecting boundary values of the eye outline; detecting amaximum value and a minimum value of the boundary values; and detectingwhether a difference between the maximum value and the minimum value ofthe boundary values decreases and then increases, and determining thatthe eye outline blinks if the difference between the maximum value andthe minimum value of the boundary values decreases and then increases.

For example, in an embodiment, whether the eye outline blinks may bedetermined by using a relevant interface function in the OpenCV, aspecific method of which may include the following steps 411 to 413.

Step 411 includes: detecting a boundary of the eye outline according toa cvMatchTemplate interface function, a specific code instruction ofwhich is as follows:

-   -   cvMatchTemplate(img, tpl, tm, CV_TM_CCOEFF_NORMED).

The tpl is an eye template created by using the cvResetImageROI(gray)interface function.

Step 412 includes: detecting, by the cvMinMaxLoc interface function, amaximum value and a minimum value of boundary values of the eye outline,a specific code instruction of which is as follows:

-   -   cvMinMaxLoc(tm, &minval, &maxval, &minloc, &maxloc, 0).

Step 413 includes: detecting whether a difference between the maximumvalue and the minimum value of the boundary values of the eye outlinedecreases and then increases, that is, determining whether the eyecloses, and determining that the eye outline blinks if the differencebetween the maximum value and the minimum value of the boundary valuesof the eye outline decreases and then increases, specific codeinstructions of which are as follows:

if (maxval < TE_THRESHOLD) return 0; // return the search window *window= win; // return eye location *eye = cvRect( win.x + maxloc.x, win.y +maxloc.y, TPL_WIDTH, TPL_HEIGHT ); if ((maxval > LB_THRESHOLD) &&(maxval < UB_THRESHOLD))

return 2; //close the eye, which is a code instruction of detecting theprocess that the difference between the maximum value and the minimumvalue of the boundary values of the eye outline decreases.

-   -   if (maxval>OE_THRESHOLD)

return 1; //open the eye, which is a code instruction of detecting theprocess that the difference between the maximum value and the minimumvalue of the boundary values of the eye outline increases.

For example, in a specific application scenario, in case that the useris having a chat by using an instant communication tool of online video,in this embodiment of the disclosure, images shot by a camera may bedisplayed in a blurring manner. For example, an image of a head of theuser shot by a camera is displayed in a video chatting picture in theblurring manner, and a web advertisement (i.e., the media information)is presented in the video chatting picture, where the web advertisementmay display advertisement content and may display a “next” button and a“close” button. In case that the head of the user moves, and the eyeoutline moves to the “next” button, the current web advertisement may beswitched to a next web advertisement; and in case that the eye outlinemoves to the “close” button, the web advertisement may be closed.Meanwhile, the web advertisement may be billed according to the periodof time in which the eye outline intersects with the web advertisement.

Corresponding to the foregoing method, an apparatus for controllingtriggering of a human-computer interaction operation is further providedaccording to the embodiments of the disclosure, to perform the foregoingmethod. The apparatus may be realized by a computer device. The computerdevice may be a personal computer, a server, or a portable computer(such as a notebook computer or a tablet computer, etc). The computerdevice may include at least one processor and a memory. The memorystores computer readable instructions, and all or part of the processesin the foregoing method embodiments of the disclosure may be implementedthrough executing the computer readable instructions by the processor.The memory may be a magnetic disk, an optical disc, a read-only memory(ROM or a random access memory (RAM), etc. FIG. 4 is a schematiccomposition diagram of an apparatus for controlling of triggering of ahuman-computer interaction operation according to an embodiment of thedisclosure. As shown in FIG. 4, the apparatus may include:

a first module 401, configured to acquire images shot by a shootingdevice, and display the images on a display screen in a translucentmanner;

a second module 402, configured to detect difference between images ineach frame, recognize a designated outline on the shot image accordingto the difference between images in each frame, and calculate a locationof the recognized designated outline on the display screen; and

a third module 403, configured to determine, in real time, whether thedesignated outline at the location on the display screen intersects witha designated area displayed on the display screen, and trigger anoperation corresponding to the designated area if the designated outlineat the location on the display screen intersects with the designatedarea displayed on the display screen.

In an embodiment, the third module 403 is specifically configured todetermine, in real time, whether the designated outline at the locationon the display screen intersects with the designated area displayed onthe display screen, and trigger the operation corresponding to thedesignated area if the designated outline on the display screenintersects with the designated area displayed on the display screen,where the triggered operation includes: recording a period of time inwhich the designated outline intersects with the designated area,determining whether the designated outline on the display screen movesout of the designated area, stopping recording the period of time if thedesignated outline on the display screen moves out of the designatedarea, and continuing recording the period of time if the designatedoutline on the display screen does not move out of the designated area.A billing operation may be further performed on the designated areaaccording to the period of time.

In another embodiment, the third module 403 is specifically configuredto: determine, in real time, whether the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen, and trigger the operation correspondingto the designated area if the designated outline at the location on thedisplay screen intersects with the designated area displayed on thedisplay screen, where the triggered operation includes: detectingwhether a designated motion form of the designated outline occurs, andtriggering an instruction operation bound with the designated area ifthe designated motion form of the designated outline occurs.

In still another embodiment, the designated outline is an eye outline,and the third module 403 detects whether the designated motion form ofthe designated outline occurs, which specifically is: detecting whetherthe eye outline blinks.

The designated outline may alternatively be an outline of other pattern,such as an outline of a human mouth. In this case, the designated motionform may be opening and closing actions of a mouth.

In yet another embodiment, the designated area displayed on the displayscreen may be a display area of designated media information, adesignated instruction area, or an area of other designated displayform.

The modules according to the embodiments of the disclosure may berealized by software (for example, a computer readable instruction whichis stored in a computer readable medium and executed by a processor), byhardware (for example, a processor of an application specific integratedcircuit (ASIC)), or by a combination of software and hardware, which isnot limited in the embodiments of the disclosure.

The modules according to the embodiments of the disclosure may beintegrated, deployed separately, combined as one module, or furtherdivided into multiple sub-modules.

FIG. 5 is a schematic hardware diagram of an apparatus for controllingtriggering of a human-computer interaction operation according to anembodiment of the disclosure. As shown in FIG. 5, the apparatus mayinclude: a processor 51, a memory 52, at least one port 53, and aninterconnecting mechanism 54. The processor 51 and the memory 52 areconnected via the interconnecting mechanism 54. The apparatus mayreceive and send data information via the port 53, where

the memory 52 stores a computer readable instruction,

the processor 51 executes the computer readable instruction to performthe following operations:

acquiring images shot by a shooting device, and displaying the images ona display screen in a blurring manner;

detecting difference between the images in each frame, recognizing adesignated outline on the shot image according to the difference betweenthe images in each frame, and calculating a location of the recognizeddesignated outline on the display screen; and

determining, in real time, whether the designated outline at thelocation on the display screen intersects with a designated areadisplayed on the display screen, and triggering an operationcorresponding to the designated area if the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen.

In this embodiment of the disclosure, the processor 51 may execute thecomputer readable instruction stored in the memory 52 to further performall or part of processes according to the foregoing method embodiments,which is not described herein.

It can be seen that in case that the computer readable instructionstored in the memory 52 is executed by the processor 51, functions ofthe first module 401, the second module 402, and the third module 401described above may be achieved.

It can be seen that in the embodiments of the disclosure, the imagesshot by the shooting device are acquired; the images are displayed onthe display screen in the blurring manner such as in a translucentmanner, so that the images may superimpose other interface displayed onthe display screen, and the designated outline (for example, an outlineof an organ such as an eye of human and a mouth of human) on the imagecan be recognized; and a user may control movement of the designatedoutline on the image by moving his body, and in case that the designatedoutline intersects with the designated area (which, for example, may bea display area for media information, or a designated instruction areasuch as a button or a link, etc) displayed on the display screen, theoperation corresponding to the designated area is triggered. Therefore,the human-computer interaction operation may be triggered without usinghands, which facilitates those with disabled hands to trigger a computeroperation in a contactless manner.

It should be noted that not all of the steps and modules in theforegoing processes and structural diagrams are necessary, and somesteps or modules may be omitted as needed. The steps may be performed indifferent orders, which may be adjusted as needed. The modules aredistinguished by function only for easy description. In actualimplementation, one module can be implemented by multiple modules, andfunctions of multiple modules can be implemented by one module. Themodules may be arranged in one device or arranged in different devices.

Hardware modules in the embodiments of the disclosure may be realized ina mechanical manner or an electronic manner. For example, a hardwaremodule may include a specially designed permanent circuit or a logicdevice (for example, an application specific processor such as an FPGAor an ASIC) to achieve a specific operation. The hardware module mayalternatively include a programmable logic device or a circuit (forexample, include a universal processor or other programmable processor)which is configured by software to perform specific operations. In themechanical manner, the hardware module is realized by using theapplication-specific permanent circuit, or using the temporarilyconfigured circuit (for example, configured by software), which may bedecided by taking account of costs and time.

Based on the foregoing descriptions of the embodiments, a person skilledin the art may clearly understand that the disclosure may be implementedby a combination of software and a necessary universal hardwareplatform, that is, the disclosure may be implemented by instructingrelated hardware by using computer readable instructions. The disclosuremay also be implemented by hardware, however, the combination ofsoftware and hardware is a preferred embodiment in most cases. Based onsuch an understanding, the essence of the technical solutions of thedisclosure, or the part of the technical solutions making contributionto the conventional technology may take a form of a software product.The computer software product is stored in a storage medium, includes aplurality of instructions, and is configured to enable a terminal device(which may be a mobile phone, a personal computer, a server, or anetwork device, etc) to perform the methods described in the embodimentsof the disclosure.

A person of ordinary skill in the art may understand that all or part ofthe processes of the foregoing method embodiments may be achieved byinstructing related hardware modules by using the computer readableinstructions. The computer readable instructions may be stored in acomputer readable storage medium, when executed by one or moreprocessors on the terminal device, may cause the terminal device toexecute the steps or functions in the described method. The processes ofthe foregoing method embodiments may be implemented by performing thecomputer readable instructions. The storage medium may be a magneticdisk, an optical disc, a read-only memory (ROM), or a random accessmemory (RAM), or in a cloud memory accessed via an Internet connectionetc. Optionally, the computer readable instructions may be downloaded bya communication network from a server computer. The computerinstructions may be hardware program codes modules constituted as logicgates microcodes within a ROM, a processor or a controller, or asfunctional modules constituted by both software/hardware together.

The foregoing descriptions are embodiments of the disclosure, but arenot intended to limit the disclosure. Any modification, equivalentalternation, or improvement made within the spirit and principle of thedisclosure shall fall within the protection scope of the disclosure. Thescope of the claims of the disclosure should not be limited to theembodiments described above, and the specification as a whole should beexplained with the widest scope.

1. A method for controlling triggering of a human-computer interactionoperation, comprising: acquiring images shot by a shooting device, anddisplaying the acquired images on a display screen in a blurring manner;detecting a difference between the acquired images in each frame,recognizing a designated outline on the acquired image according to thedifference between the acquired images in each frame, and calculating alocation of the recognized designated outline on the display screen; anddetermining, in real time, whether the designated outline at thelocation on the display screen intersects with a designated areadisplayed on the display screen, and triggering an operationcorresponding to the designated area if the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen.
 2. The method according to claim 1,wherein in case that the designated outline at the location on thedisplay screen intersects with the designated area displayed on thedisplay screen, the triggered operation corresponding to the designatedarea, comprises: recording a period of time in which the designatedoutline intersects with the designated area; determining whether thedesignated outline on the display screen moves out of the designatedarea; stopping recording the intersection time if the designated outlineon the display screen moves out of the designated area, and continuingrecording the period of time if the designated outline on the displayscreen does not move out of the designated area.
 3. The method accordingto claim 2, wherein the method further comprises: calculating billinginformation corresponding to the designated area according to therecorded period of time in which the designated outline intersects withthe designated area.
 4. The method according to claim 1, wherein in casethat the designated outline at the location on the display screenintersects with the designated area displayed on the display screen, thetriggered operation corresponding to the designated area comprises:detecting whether a designated motion form of the designated outlineoccurs; and triggering an instruction operation bound with thedesignated area if the designated motion form of the designated outlineoccurs.
 5. The method according to claim 4, wherein the process ofdetecting whether a designated motion form of the designated outlineoccurs comprises creating a template of the designated outline;detecting images in each frame in the template of the designatedoutline; and determining whether a change between the acquired images ineach frame satisfies the designated motion form, and triggering theinstruction operation bound with the designated area if the changebetween the acquired images in each frame satisfies the designatedmotion form.
 6. The method according to claim 4, wherein the designatedoutline is an eye outline; and the process of detecting whether adesignated motion form of the designated outline occurs comprises:detecting whether the eye outline blinks.
 7. The method according toclaim 5, wherein the designated outline is an eye outline; and theprocess of detecting whether a designated motion form of the designatedoutline occurs comprises: detecting whether the eye outline blinks. 8.The method according to claim 6, wherein the process of detectingwhether the eye outline blinks comprises: detecting boundary values ofthe eye outline; detecting a maximum value and a minimum value of theboundary values; and detecting whether a difference between the maximumvalue and the minimum value of the boundary values decreases and thenincreases, and determining that the eye outline blinks if the differencebetween the maximum value and the minimum value of the boundary valuesdecreases and then increases.
 9. The method according to claim 7,wherein the process of detecting whether the eye outline blinkscomprises: detecting boundary values of the eye outline; detecting amaximum value and a minimum value of the boundary values; and detectingwhether a difference between the maximum value and the minimum value ofthe boundary values decreases and then increases, and determining thatthe eye outline blinks if the difference between the maximum value andthe minimum value of the boundary values decreases and then increases.10. The method according to claim 1, wherein the designated areadisplayed on the display screen is a display area of designated mediainformation or a designated instruction area.
 11. The method accordingto claim 2, wherein the designated area displayed on the display screenis a display area of designated media information or a designatedinstruction area.
 12. The method according to claim 3, wherein thedesignated area displayed on the display screen is a display area ofdesignated media information or a designated instruction area.
 13. Themethod according to claim 4, wherein the designated area displayed onthe display screen is a display area of designated media information ora designated instruction area.
 14. The method according to claim 5,wherein the designated area displayed on the display screen is a displayarea of designated media information or a designated instruction area.15. An apparatus for controlling triggering of a human-computerinteraction operation, wherein the apparatus comprises at least oneprocessor and a memory having processor-executable instructions storedtherein, and the instructions when executed by the at least oneprocessor, configure the apparatus to: acquire images shot by a shootingdevice, and display the acquired images on a display screen in ablurring manner; detect a difference between images in each frame,recognize a designated outline on the shot image according to thedifference between acquired images in each frame, and calculate alocation of the recognized designated outline on the display screen; anddetermine, in real time, whether the designated outline at the locationon the display screen intersects with a designated area displayed on thedisplay screen, and trigger an operation corresponding to the designatedarea if the designated outline at the location on the display screenintersects with the designated area displayed on the display screen. 16.The apparatus according to claim 15, wherein the apparatus is furtherconfigured to: trigger the operation corresponding to the designatedarea in case it is determined that the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen, wherein the triggered operationcomprises the apparatus configured to: record a period of time in thedesignated outline intersects with the designated area; determinewhether the designated outline on the display screen moves out of thedesignated area; stop recording the period of time if the designatedoutline on the display screen moves out of the designated area, andcontinue recording the period of time if the designated outline on thedisplay screen does not move out of the designated area.
 17. Theapparatus according to claim 15, wherein the apparatus is furtherconfigured to: trigger the operation corresponding to the designatedarea in case it is determined that the designated outline at thelocation on the display screen intersects with the designated areadisplayed on the display screen; wherein the triggered operationcomprises the apparatus configured to: detect whether a designatedmotion form of the designated outline occurs, and trigger an instructionoperation bound with the designated area, if the designated motion formof the designated outline occurs.
 18. The apparatus according to claim17, wherein the designated outline is an eye outline; and the apparatusis further configured to detect whether the eye outline blinks.
 19. Theapparatus according to claim 15, wherein the designated area displayedon the display screen comprises a display area of designated mediainformation or a designated instruction area.
 20. The apparatusaccording to claim 16, wherein the designated area displayed on thedisplay screen comprises a display area of designated media informationor a designated instruction area.